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THE INDEPENDENT VERIFICATION, VALIDATION, AND ACCREDITATION PROCESS AS 
APPLIED TO EXTENDED AIR DEFENSE TESTBED 


Kathy Selvidge 
Quality Research, Inc. 
Huntsville, Alabama 


Abstract 

This paper discusses the Independent 
Verification, Validation and Accreditation 
Process as applied to Extended Air Defense 
Testbed (EADTB). The process covers logical 
verification and code verification techniques and 
both structural validation and output validation 
techniques. Specific examples from EADTB 
are included to demonstrate application of the 
techniques. 

1.0 Background 

The focus of the EADTB design is to support 
analyses that encompass all aspects of the joint 
service extended air defense issues for Theater, 
Air and Missile Defense (TAMD) with emphasis 
on Battle Management Command, Control, 
Communications and Computer (BMC4I) 
architecture and operations. The EADTB 
provides common tools for supporting analyses 
of: present and evolving air threat, present air 
and space based defense effectiveness and 
limitations, and future conceptual defenses 
involving operational and/or technological 
improvements. The EADTB is used by materiel 
developers to define and evaluate system 
concepts, by combat developers to develop 
doctrine and tactics, by the Tester and Evaluator 
(T&E) to support testing, and by operational 
commanders for staff training and battle 
planning (such as EADTB’s support of Roving 
Sands). 

The EADTB provides for the simulation of 
scenarios ranging from few-on-few to theater 
level (and extendable to NMD levels) using a 
common set of tools and algorithms. EADTB is 
being designed to be compliant with the Defense 
Modeling and Simulation Office (DMSO) High 
Level Architecture (HLA) rules. The EADTB 
represents land, sea, air, and space systems in 
active defense, passive defense, attack 
operations, and BM/C4I as well as the 
environment, other targets, and the interactions 
with other forces and missions. 


EADTB models are composed of user provided 
data feeding generic algorithms (with modifiable 
rulesets to govern the control structure) which 
can be configured into models or Specific 
System Representations (SSRs). These 
representations along with an experiment 
preparation system form the basis for a library 
system enabling analyses to be constructed, 
calibrated, and varied more rapidly. 

2,0 Executive Summary 

This paper presents a methodology for 
verification, validation and accreditation of 
models and simulations. This paper contains 
information for conducting a complete 
verification, validation and accreditation study, 
from the problem definition stage to the 
documentation of the results in a comprehensive 
final report. Attention to detail is paramount 
throughout the VV&A effort to ensure that the 
final product meets the needs of the customer or 
application sponsor. 

This document does not discuss every possible 
technique used in VV&A work but rather, 
presents generic processes used by Quality 
Research on the EADTB. This process can be 
expanded as programs mature and additional 
case studies become available. For details about 
VV&A techniques not mentioned herein, the 
reader is referred to the Department of Defense 
(DoD) Verification, Validation, and 
Accreditation (VV&A) Recommended Practices 
Guide. 


3.0 Key Terms 

Verification 

Verification is the process of determining that a 
model or simulation implementation accurately 
represents the developer’s conceptual description 
and specifications. (Is the simulation what I 
intended?) 
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There are two basic types of verification. 
Logical verification ensures that the basic 
equations, algorithms and logic flow are correct. 
Code verification ensures that these 
representations have been correctly implemented 
in the code. 

Verification also evaluates the extent to which 
the model or simulation has been developed 
using sound and established software 
engineering techniques. Verification is applied 
at each stage to ensure that the products of that 
stage accurately implement the specifications 
from the previous stage. 

Validation 

Validation is the process of determining the 
degree to which a model or simulation is an 
accurate representation of the real world from the 
perspective of the intended uses of the model or 
simulation. 

Validation is viewed from multiple perspectives, 
the most important of which include: 1) 
structure and depth sufficient to adequately 
represent the real world for a given application, 
2) behavior, including the ability to predict 
system performance, and 3) information 
conveyance. Validation is ultimately 
accomplished by testing and evaluating the 
results of exercising the systems and simulations 
in realistic applications 

Accreditation 

Accreditation is an official determination that the 
simulation as a whole or any one of its 
components when examined independently is 
acceptable for use for a specific purpose. 
(Should my organization endorse this 
simulation?) 

A^reditation is a decision that is based on 
several different factors, including verification 
and validation. 


4.0 W&A Activities 

The W&A activities for EADTB are closely 
tied to the life-cycle of the software development 
process for EADTB. Figure 4.0-1 shows the 
EADTB W&A Process Model. W&A 
activities are ongoing processes throughout the 
life cycle of the EADTB. W&A activities are 


accomplished at each life cycle stage of the 
EADTB. 



Figure 4.0-1 EADTB W&A PROCESS 
MODEL 

The W&A Process for EADTB is; (1) 
compliant with evolving guidance (multi-service 
and DoD) and state of practice, (2) tailored to the 
circumstances of the EADTB development 
process, simulation characteristics and intended 
uses and, (3) pro-active in building the audit trail 
of evidence necessary and sufficient for 
confidence in EADTB results and outputs. 

5.0 Verification Methods 

The general process model shown in Figure 4.0- 
1 used to perform W&A has been implemented 
by the EADTB W&A team. Figure 5.0-1 shows 
how the EADTB W&A team employs the 
process to perform verification. 



FIGURE 5.0-1 VERIFICATION PROCESS 


MODEL 

The following sections show how the W&A 
team implements this process model for the 
EADTB. 
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5.1 Requirements Analysis 

Requirements analysis involves verifying 
complete, consistent and accurate requirements, 
reviewing requirements documentation, 
verifying testability of requirements and 
establishing traceability of all requirements to 
and from a source. 

Requirements traceability helps to ensure that all 
requirements have a well-defined source and 
purpose. Traceability reduces the potential for 
modification of requirements. In addition, 
setting up traceability provides an additional 
check on the consistency and accuracy of the 
requirements. 

Requirements traceability analysis is performed 
to track and correlate requirements specifications 
from the technical requirements documentation 
through the implementation phase of the 
software development life-cycle. By tracking 
each system level requirement through each 
development phase, requirements traceability 
analysis provides an assessment of the 
completeness of the development at each phase. 

The EADTB W&A team reviews and analyzes 
all EADTB software requirements 
documentation. This review identifies: 

• critical requirements 

• design and test product items in critical 
requirements threads 

• significant omissions, inconsistencies, 
ambiguities and errors in critical design and 
test product items 

• critical requirements based on performance, 
functionality, error recovery, sizing and 
timing constraints or other criteria 

• CSUs, CSCs, CSCIs, interfaces, interface 
messages, test cases and test procedures 
associated with critical requirements 

The items identified from the documentation 
reviews lead to the determination that: 

• the requirements documentation adheres to 
the applicable standards 

• the system fits correctly within its global 
context and all external originating 
requirements are adequately incorporated or 
otherwise accommodated 

• each requirement is valid, adequate, correct 
and unambiguous 


• all requirements correctly trace to all 
appropriate levels of specifications and that 
the references are re-verified whenever a 
change occurs. 

The EADTB W&A team developed a tool to 
perform requirements analysis. The tool consists 
of a database that contains all of the software 
requirements associated with the EADTB 
framework. Keyword searches allow the tester 
to easily review the database for any requirement 
associated with the test. The tester also has the 
ability to review requirements associated with 
the unit under test and generate reports 
containing these requirements. Once the 
requirements are tested, the database provides a 
repository where EADTB’s adherence to the 
requirements may be tracked. An area is 
provided to describe any software change 
requests or trouble reports associated with the 
tested requirement. All the data is recorded for 
retrieval in a variety of reports and is related to 
the test name for archival purposes. Figure 5.1-1 
shows the main testing table used to track 
requirements and software trouble reports for 
EADTB. 



Figure 5.1-1 W&A Testing Database 


5.2 Algorithm Analysis 

An in-depth analysis is performed on the 
EADTB algorithms to verify that each major 
algorithm is mathematically correct and 
consistent with established mathematical 
practices. This involves rigorous verification of 
the mathematics of an algorithm to ensure that 
the equations are derived correctly and that there 
are no errors in the expressions. 

Algorithm analysis must be a part of logical 
verification, code verification and testing. The 
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initial analysis of the algorithms occurs during 
documentation reviews. This includes the 
mathematical analysis mentioned above and is a 
crucial part of logical verification. Figure 5.2-1 
shows the approach to algorithm analysis 
followed by the EADTB W&A team. 



FIGURE 5.2-1 ALGORITHM ANALYSIS 


Algorithm evaluation is a crucial part of logical 
verification. The algorithms are evaluated 
during documentation reviews to ensure that the 
appropriate algorithm is provided for each of the 
requirements. Alternative algorithms may be 
suggested to the developer at this point. Each of 
the major algorithms is analyzed for adherence to 
established mathematical and physical theory. 
The major equations in the algorithms are 
derived from first principles to establish 
correctness. Any errors found in the derivations 
of the algorithms are presented to the developer 
for correction before implementation. The 
mathematical expressions within the algorithms 
are verified to be correct and free from errors. 
Mathematical analysis tools are used by the 
W&A team for this purpose. The algorithms 
may be modeled in a suitable simulation or 
stand-alone for analysis and comparison when 
appropriate. 

Sensitivity Analysis 

Sensitivity analyses are performed to ensure that 
EADTB is reacting to varied inputs in an 
expected and predictable manner. Input data are 
systematically varied to test the reaction of the 
simulation to changes. Extreme and boundary 
layer testing is performed to determine reaction 
of EADTB to stressing conditions. Sensitivity 
analysis facilitates the verification process by 
highlighting the effects of input data changes on 
the functional outputs of the code. An EADTB 


specific example follows which shows the results 
of a sensitivity analysis performed within a 
tracking function. 

Effect of Process Noise on Tracking 

Within the ruleset call to the filtering algorithm, 
the user may specify a process noise value as one 
of the arguments of the hard-coded algorithm. 
This value is actually the sigma acceleration used 
to build the process noise covariance matrix. 
The process noise covariance plays an important 
role in the successful tracking of an object. The 
process noise is an impulse noise on 
acceleration. The filter will trust the 
extrapolation model believing it to be a good 
model of the target dynamics, if the process 
noise is low. The filter will believe the sensor 
measurement if the process noise is high. It is 
necessary to choose this value carefully. If the 
process noise is set too low, the filter becomes 
unresponsive to the sensor measurement. If the 
process noise is set too high, it may cause 
divergence. 

It is possible to define a high value for the 
process noise and maintain a track on a 
maneuvering air-breathing threat, ABT. Even 
though the kinematic model is a constant 
velocity model, because of the high process 
noise the filter will trust the sensor measurement 
more than the kinematic model and it will weight 
the estimate accordingly. 

A scenario was created to stress the ability to 
track a maneuvering aircraft with a constant 
velocity Kalman filter. The process noise 
argument in the call statement to the hard-coded 
algorithm for filtering an ABT state was varied 
on sequential runs. The first set of runs tested 
the ability to track the maneuvering target with a 
low process noise. The second set of runs tested 
the ability to track the maneuvering target with a 
high process noise. 

It was expected that the first set of runs would 
produce much larger track errors or even losses 
of track because of the inability to track during 
maneuver. The high process noise in the second 
set of runs would compensate for the 
acceleration within the filter to allow it to 
continue tracking the target even when it was in 
maneuver. 

Figure 5.3-1 shows the history trail or true 
position and the perception trails as seen by the 
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rear radar for a maneuvering target using low 
process noise covariance. The result is that the 
target is not tracked as well in maneuver as it is 
in the constant velocity legs of the waypoint set. 
The perception trails are seen diverging from the 
actual position. This filter believes that the 
model provides good estimates of the target’s 
position. Therefore, when the target begins to 
maneuver, the filter expects that the target will 
be continuing on its constant velocity heading. 

If a high process noise covariance is 
implemented within the filter, the measurement 
from the sensor becomes much more important. 
The radar is better able to track the target with its 
continual updates from the radar and because the 
filter believes the sensor measurements more 
than the model of the target dynamics. Figure 

5.3- 4 shows that the perception trail and the 
history trail differ only slightly. 

EADTB provides the ability through ruleset 
interaction to change the process noise 
covariance matrix and obtain different tracking 
characteristics. A comparison of Figure 5.3-2 
and 5.3-5 shows that the actual position error is 
greatly diminished when the process noise 
covariance matrix is increased. Similarly the 
actual velocity error shown in Figures 5.3-3 and 

5.3- 6 illustrates that the high process noise 
covariance matrix allows for better actual 
velocity track errors of the maneuvering target. 



FIGURE 5.3-1 MANEUVERING TARGET 
TRACKED WITH LOW PROCESS NOISE 
INPUT 



FIGURE 5.3-2 ACTUAL POSITION ERROR 
FOR HIGH MANEUVERING, LOW PROCESS 
NOISE TARGET TRACK 



FIGURE 5.3-3 ACTUAL VELOCITY ERROR 
FOR HIGH MANEUVERING, LOW 
PROCESS NOISE TARGET TRACK 



FIGURE 5.3-4 MANEUVERING TARGET 
TRACKED WITH HIGH PROCESS NOISE 
INPUT 
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FIGURE 5.3-5 ACTUAL POSITION ERROR 
FOR HIGH MANEUVERING, HIGH 
PROCESS NOISE TARGET TRACK 



FIGURE 5.3-6 ACTUAL VELOCITY ERROR 
FOR HIGH MANEUVERING, HIGH 
PROCESS NOISE TARGET TRACK 


Code assessments are used to assess the 
complexity, adherance to coding standards and 
recommended practices, imbedded faults, and 
structural errors. Any inconsistencies are 
identified between the Program Notebook (PNB) 
and the code. The EADTB W&A team uses 
code assessments to evaluate threads through 
calling sequences for selected events as shown in 
Figure 5.4-1. 


CMS__SN__PCM_Controls. Event_Dispatcher 

Execute_Start_Search 

CMS__SN_MDL__Radar. Start_Search 

CMS_SN_DT_Radar_Control_Mode.Long_ 
Term_Cost_Of 

CMS_SN_PCM_Commands.Active_Scan_ 
Complete 

Compute_Next_Activation_Time 

CMS_SN_PCM_Events_Manager.Schedule 
("Active_Scan_Complete") 

FIGURE 5.4-1 CALL SEQUENCES USED 
WHEN PERFORMING CODE 
ASSESSMENTS 

The call threads enable the tester to determine 
which coding packages are affected by errors in 
implementation of the algorithms. The code 
assessment helps determine the impact of errors. 


5.4 Code Assessments, Code Inspections and 
Walk-Throughs 

Code Assessments, code inspections and walk¬ 
throughs are performed by the EADTB W&A 
team to determine if the algorithms are 
implemented properly within the code. Code 
inspections ensure consistency, correctness and 
completeness in the implementation. 

Within EADTB, code inspections are conducted 
primarily using the code debugger tool. Code 
debugging allows the tester to examine the code 
while stepping line-by-line through the 
algorithms of interest. Output may be written to 
the screen for visual inspection or written to a 
log file for future examination of the data. 
Discrepancies between accepted equations and 
algorithms and the implemented code are 
reported. 


Testing 

EADTB requires user-defined rulesets to invoke 
specific symbols which, are in turn, used to 
stimulate hard-coded algorithms within the ADA 
code. Testing seeks to construct the necessary 
test scenario to adequately examine the area of 
interest. Within EADTB, this requires that the 
tester prepare system models by defining the 
necessary input parameters and ruleset calls. 
The tester must define the gameboard to be used, 
deploy the scenario elements, build message data 
tables and network connectivity data sets, resolve 
aliases and identify measures of effectiveness 
(MOEs) and measures of performance (MOPs) 
for output. 

Once developed, the scenario is executed within 
an experiment. Data for a run is archived in 
trace logs, error logs and recorded data items for 
future examination. A graphical user interface is 
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available to inspect the operations of a scenario. 
Radar detection/tracking lines, radar modes, 
message communications, platform movement, 
perception lines, history trails and entity states 
may be examined by inspection. 

MOEs and MOPs are recorded through an 
analysis tool. This tool allows for playback of 
the scenario as well as examination of the 
recorded data items. These may be exported to 
for further analysis using spreadsheets or parsing 
codes. An example follows of a test that was 
performed to test the number of hops to a 
gateway requirement within EADTB. 

Number of Hops to Gateway 

A gateway component is a component that is 
designated as a message translator from one 
network to another network. The use of the 
gateway is utilized when a message is set to 
transmit to a Comm component on the other 
network. The design of the networks for a 
gateway is shown in Figure 5.5-1 below. 



Figure 5.5-1 Design of the Networks for a 
Gateway 


EADTB states that Comm shall determine the 
maximum number of hops for retransmission on 
the new network based on the following 
parameters: Number of hops that the message 
used to get to the Gateway and maximum 
number of hops for the original message. The 
Figure 5.5-1 illustrates the use of a gateway 
exactly four hops away from the message 
transmitter on the sending network. The 
message is sent to the ScenEl on the receiving 
network that has line of sight with the gateway. 
As Figure 5.5-1 illustrates, the message will take 
a minimum of four hops to get to the destination. 
The Comms Link Status RCD shows the various 
link connectivity status enumerations for the 
ScenEls per network. The valid paths can be 


traced from the Comms Link Status RCD as 
shown in Table 5.5-1 below. 


Table 5.5-1 Comms Link Status RCD 


Slants 

Time 

I ink 
ID 

Network 

ID 

Source 
Seoul 1 
Comm 
Comp 
ID 

Destination Connect i\ il> 
ScenEl Status 

Comm 

Comp ID 

HEEffli] 

i 

i 


339123 


(fttiingsmi 

2 

i 


339126 


0.00E+00 

3 

i 

339228 

339123 

OPERABLE 

0.00E+00 

4 

i 

339123 

339228 

OPERABLE 

0.00E+Q0 

5 

i 

339228 

339126 

OPERABLE 

0.00E+00 

6 

i 

339126 

339228 

OPERABLE 

0.00E+00 

1 

2 

338794 

338791 

OPERABLE 

0.00E+00 

2 

2 

338791 

338794 

OPERABLE 

0.00E+00 

3 

2 

339211 

338791 

OPERABLE 

0.00E+00 

4 

2 

338791 

339211 

OPERABLE 

O.OOE+OO 

5 

2 

339227 

338791 

NON LOS 

0.00E+00 

6 

2 

338791 

339227 

NON LOS 

0.00E+00 

7 

2 

339211 

338794 

NON LOS 

0.00E+00 

8 

2 

338794 

339211 

NON LOS 

0.00E+00 

9 

2 

339227 

338794 

NON LOS 

0.00E+00 

10 

2 

338794 

339227 

NON LOS 

0.00E+00 

11 

2 

339227 

339211 

OPERABLE 

0.00E+00 

12 

2 

339211 

339227 

OPERABLE 


6.0 Validation Methods 

According to DoD 5000.59, validation is the 
rigorous and structured process of determining 
the extent to which modeling and simulation 
accurately represents the intended “real world” 
phenomena from the perspective of the intended 
use of the model and simulation. Validation has 
two main components: structural validation and 
output validation (also called conceptual model 
validation and results validation). Structural 
validation focuses on the internal portion of the 
modeling and simulation which includes 
examination of modeling and simulation 
assumptions and review of the modeling and 
simulation architecture and algorithms in the 
context of their intended use. Output validation 
answers questions on how well the simulation 
results compare with the perceived real world. 

The Validation Process used for EADTB is 
illustrated in Figure 6.0-1 and is comprised of 
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four main tasks: (1) problem definition, (2) 
structural validation, (3) output validation, and 
(4) preparation of a validation report. These 
elements are discussed in the following sections. 



FIGURE 6.0-1 VALIDATION PROCESS FOR 
EADTB 


6.1 Problem Definition 

The validation process begins with a clear and 
unambiguous statement or definition of the 
problem. A good definition of the problem 
makes it easier to define its solution 
requirements. These requirements are the 
features, characteristics, or functions that are 
important to the problem and essential to its 
solution. Two pillars compose the problem 
definition process. The first is the identification 
of the real world being modeled. The second is 
the identification of the key structural 
characteristics and output parameters that are to 
be used for the comparisons during the 
validation process. The procurement of standard 
data has also been included in the problem 
definition process because the outcome of the 
first two steps is a clear understanding of the data 
input items that are required by the simulation. 

6,1.2 Identification of the Real World 

Identification of the real world involves 
definition of the system(s) that will be modeled 
and to what level of fidelity (i.e. terrain, weather, 
environment, features). Validation is the 
comparison of the M&S behavior and results to 
data obtained from another credible domain that 
is either believed to be the real world, has been 
proven to closely approximate the real world, or 
is from a source that is recognized as expert on 
the relevant characteristics of the real world. 
Some real world data sources include the 
following. Each of these real worlds has 
inherent drawbacks and limitations that can 
make or break the validity of a simulation. 


• Subject matter experts (SME) or other 
recognized individuals in the field of 
inquiry. Face validation enables the 
“experts” to compare the simulation 
structure and output to their estimation of 
the real world. • 

• Scientific theory and accepted algorithms 
that define the ranges of acceptable behavior 
in response to given inputs. 

• Laboratory test, developmental test, system 
operational test or other engineering data 
that provide a set of empirical data points 
that correspond to specifically identified 
input data. 

• Training facility measurements and live fire 
training and test results for comparison. 

• Comparison with historical values. This 
includes measurements of the phenomena of 
war or historical results. 

• Touchstone modeling and simulation such 
as previously and separately validated 
models and simulations. 

The W&A team participated in a validation of 
the ballistic missiles modeled within EADTB. 
The U.S. Army Space and Missile Defense 
Command (SMDC) has developed a family of 
targets that is representative of a variety of 
threats to allow a comprehensive evaluation of 
theater missile defense system capabilities. 
Models of these targets and other ballistic 
missile systems were constructed in the 
Extended Air Defense Testbed (EADTB). These 
models were validated to ensure they are an 
accurate representation of the real system. The 
purpose of the validation effort was to provide a 
quantitative assessment of how well the models 
represent the real systems. The following are the 
targets and other ballistic missile systems which 
were constructed and validated in EADTB: 

• MBRV-1 Single Stage and Two Stage 
Configurations 

• Two Stage Hera Block IITHAAD Unitary 
Target 

• Two Stage Hera Block II Patriot Unitary 
Target 

• Multi-Service Launch System (MSLS) 

Target 

• Aries 


UNCLASSIFIED 

8 













UNCLASSIFIED 


• Lance GM-52C Missile 

The purpose of the validation effort was to 
construct models of these ballistic missile 
systems and then compare the models and their 
behavior to the real systems and their behavior. 
There are two distinct methods for representing 
ballistic trajectories in EADTB: 1) EADTB 
internally-generated trajectories, and 2) imported 
pre-planned threat tape trajectories. All of the 
testing was conducted using the EADTB 
internally generated trajectories. Ballistic Missile 
Move (BMM) is a collection of algorithms 
within EADTB which provide a variety of 
ballistic motion options. Each ballistic missile 
scenario element (ScenEl) has input parameters 
which are established by an EADTB user, stored 
in a relational database, and retrieved and loaded 
into BMM at the start of an experiment. These 
parameters can be broken into three types: 

a. Specific System Representation (SSR) 
parameters which specify the 
performance capabilities of ballistic 
missiles of a given type. 

b. Instance parameters which are unique 
for a given ballistic missile scenel and 
do not change throughout the 
experiment. 

c. Dynamic parameters which are unique 
for a given ballistic missile scenel, are 
initialized by the user, and can change 
as the experiment is executed. 

The values for these parameters were derived 
from missile performance and flight test data 
provided by the Targets Office for each of the 
systems listed above. 

Key Structural Characteristics and Output 
Parameters 

Identification of key structural characteristics 
and output parameters involves determining the 
categories that are of interest for the intended use 
of the model or simulation. Examples of key 
structural characteristics are mass, dimensions, 
thrust, bum rates, available power, location, 
terrain, weather, and backgrounds. Key output 
parameters are used to assess the validity of the 
model. Examples of these are flight trajectory 
(range, apogee, time of flight), number of 
missiles launched, number of missiles killed, and 
message error rates. These parameters may be 


found in various MOEs, MOPs, and Recorded 
Data Items (RCDs) and are illustrated in Figure 
6.1.3-1. 


• Key Structural Characteristics 

- Engine Thrust 

- Engine Start and Stop Times 

- Mass Burn Rate 

- Platform, Payload, and Fuel Mass 

- Configuration and Dimensions 

- Trajectory Requirements (Range and Apogee) 

* Key Output Parameters 

- Trajectory Profile 


FIGURE 6.1.3-1 ILLUSTRATION OF KEY 
STRUCTURAL CHARACTERISTICS AND 
OUTPUT PARAMETERS 

Emphasis in different areas dictates that different 
items are of importance. The following 
categories of models and simulations illustrate 
this difference: 

• Research and Development. These 
simulations require a high level of fidelity. 
Validation emphasizes completeness and 
balance of algorithms. Items of importance 
include the portrayal of the subsystems, 
components and system parameters, physics 
phenomena, and interactions with the 
environment. 

• Education and Training. These simulations 
provide emphasis on education and training 
of soldiers. As such, validation items will 
tend to center around human interactions 
and interfaces and the quality of after action 
reviews. 

• Analysis. These simulations support the 
analysis of weapon system effectiveness, 
new tactics and doctrine, force structure 
studies, and budget assessments. Validation 
centers on issues of algorithm’s robustness, 
completeness and balance. 
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Once the real world system has been defined and 
the key structural characteristics and output 
parameters have been identified, the analyst 
should have a clear understanding of the data 
input items that are required by the model. 
These items will include both data used to 
construct the model as well as data used to 
evaluate the output. The analyst is responsible 
for obtaining standard data for this purpose. 
There is a distinction in terminology here when 
referring to data that have been verified, 
validated, and certified. Verified data are 
useable by the M&S code. Validated data have 
been reviewed for their reasonableness and 
conformance to real world when available. 
Certified data come from approved data sources. 
Prior to obtaining any data, the analyst shall 
confirm whether or not certified data are 
required. 


1. Data Certification: If the customer or 
application sponsor requires certified data from 
a data supplier, then the analyst must request a 
new set of certified data for each application of 
the M&S. Threat data must be reviewed and 
approved by the appropriate intelligence agency 
prior to each use to determine whether changes 
have occurred that require updating. 
Certification establishes that the data are 
suitable for a specific use. Once the certified 
data items are received, they must be verified 
and validated. 


2. Data Verification: Data verification is the 
process of ensuring that primary source data to 
be used in the validation effort are converted to 
the correct input formats and units of measure 
and have values within the allowable range as 
specified in the design of the M&S. This 
ensures that the input and output data are in the 
proper format to be manipulated by the M&S. 
Data verification establishes that the data 
produced conform to the specification. 

3. Data Validation: Data validation is the 
comparison of input data to the corresponding 
known real world or best estimate values. Data 
validation is typically performed by the M&S 
user to ensure that the data utilized in the M&S 
are appropriate and reasonable for its usage. 


Table 6.1.4-1 


Sample Single Stage Missile Parameters 



600 km 
Long 

Range 

Ballistic 

Trajectory 

660 km 
Unballasted 
Max Range 
Trajectory 

LUT-1 

Trajectory 

Booster 

Engine 

SR-19 

SR-19 

SR-19 

Average 

Thrust 

(Newtons) 

247,597 

247,597 

247,597 

Engine Start 

Time 

(seconds) 

0.0 

0.0 

0.0 

Engine Stop 

Time 

(seconds) 

65.0 

65.0 

65.0 

Mass Bum 
Rate (kg/sec) 

95.97 

95.97 

95.97 

Specific 

263.26 

263.26 

263.26 


Impulse 

(seconds) 


Platform 

Mass (kg) 

1579.0 

1579.0 

1579.0 

Ballast Mass 
(kg) 

771.0 

0.0 

(unballasted) 

771.0 

Fuel Mass 
(kg) 

6238.0 

6238.0 

6238.0 

Payload 

Mass (kg) 

581.0 

581.0 

581.0 

Total Launch 
Mass (kg) 

9169.0 

8398.0 

9169.0 





Range (km) 

600.0 

660.0 

320.0 

Apogee (km) 

186.0 

191.0 

236.0 





Aero 

1.973 

1.973 

1.973 


Reference 
Area (m 2 ) 


Table 6.1.4-1 represents some of the data that 
was available for the systems of interest to the 
TBM validation study. 
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Preparation of Test Scenario and Structural 
Validation 

Construct Scenario 

The data collected in section 6.1.4 are used to 
construct a scenario designed to represent the 
real world as closely as possible. The validation 
study may call for using specific models or 
simulations as they are, modifying existing 
models and simulations, or developing new 
models or simulations. The primary steps for 
preparing an experiment in EADTB are 
discussed in the following section. 

6.2.I.1 EADTB 

EADTB is a high detail/fidelity simulation 
system for missile warfare analysis. The key 
steps associated with the experiment preparation 
process in EADTB are shown in Table 6.2.1-1. 
The analyst defines inputs available, outputs 
required, accuracy required, and any sensitivities 
(output-to-input) that must be correctly 
represented. The key output parameters which 
will be used to assess the validity of the model 
are often found in various MOEs, MOPs, and 
Recorded Data Items (RCDs). The analyst must 
specify which of these parameters should be 
recorded. 


Table 6.2.1.1-1 Key Steps of Experiment 
Preparation in EADTB _ 


Step 

Description 

Prepare 

System 

Models 

Select existing models from the 
master library or develop new 
ones. The model structure is 
normally divided into the four 
functional modules: Thinker, 
Platform, Sense, and 

Communicate. 

Define 

Gameboard 

Select from an existing gameboard 
definition or develop a new one. 
Gameboard definition requires 
specification of location, terrain, 
weather, and backgrounds. 

Laydown 

ScenEls 

The process of defining a 
simulated entity (referred to as a 
ScenEl) includes deploying a copy 
of a system model onto the 
gameboard, initializing dynamic 
variables, setting any static 
variables, defining any links, and 
resolving all the external aliases. 


Build 

Message Data 
Table 

A message data table defines 
communication protocols and data 
fields. An existing table can be 
used or a new table can be defined. 

Build 

Connectivity 
Data Set 

The connectivity data set defines 
communication links between 
nodes in networks and gateways 
between networks. 

Resolve 

Remaining 

Aliases 

The model code uses surrogate 
symbols for other interacting 
entities that must be specified (or 
resolved) when a ScenEl is placed 
on the gameboard. These 

surrogates, referred to as aliases, 
are partially resolved by the 
connectivity data set. EADTB 
prompts the user for any other 
unresolved aliases during 

experiment preparation. 

Specify 

MOEs and 
MOPs 

The user can select from a set of 
MOEs and MOPs that can be 
recorded during execution. In 
addition, the user can specify 
which internal dynamic variables 
(e.g. state variables) should be 
recorded. 

Specify 

Console 

Configuration 

The user specifies the 

configuration of the terminals for 
observers / participants in 
interactive runs. 

Set Execution 
Options 

EADTB offers a number of 
execution options including 

interactive, batch, and real-time. 
Experiment preparation includes 
the input of experiment start and 
stop times, preselected pause 
points, and numbers of runs in a 
Monte Carlo set. 


A scenario was constructed in EADTB for 
validating the TBM model as shown in Figure 
6.2.1.1-1. The scenario elements were 
constructed. Desired apogee and target range are 
identified at the instance level. EADTB 
internally generates the trajectory using a 3DOF 
Model. Measures of performance were recorded. 
These included the following: 

1. Altitude vs. Time 

2. Altitude vs Downrange 

3. Velocity vs Time 
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FIGURE 6.2.1.1-1 SCENARIO 
CONSTRUCTION FOR TBM VALIDATION 
INEADTB 


Structural Validation 

Structural validation focuses upon the internal 
portion of the modeling and simulation which 
includes examination of modeling and 
simulation assumptions and review of the 
modeling and simulation architecture and 
algorithms in the context of their intended use. 
The DoD W&A Recommended Practices Guide 
calls this stage “conceptual model validation” 
and defines it as the determination (usually by a 
group of subject matter experts) that the 
assumptions underlying the proposed conceptual 
model are correct and that the proposed 
simulation design elements and structure (i.e. the 
simulation’s functions, their interactions, and 
outputs) likely will lead to results realistic 
enough to meet the requirements of the 
application. Conceptual model validation 
ensures that the proposed conceptual model (and 
its resultant design) satisfies the fidelity, 
accuracy, or credibility requirements imposed by 
the specifics of the problem. Structural 
validation contains answers to the following 
questions. 


equation and algorithm methodology to 
outside documentation. Comparison to 
other accepted methodology is also possible. 

• Is there a balance of representation across all 
model and simulation components? Are 
elements of the simulation represented at 
different levels of fidelity within the 
simulation itself? What impact does this 
have on other systems within the 
simulation? What relationships exist 
between individual elements that require 
similitude in modeling levels? What aspects 
do not have to be modeled at as high a level 
of fidelity without impacting other elements 
within the simulation? (Consistency of the 
modeled systems within the scenario) 

• Does an adequate and consistent 
representation of terrain and environment 
exist across all model and simulation 
components? (Consistency of terrain and 
environment across the simulation) 

Output Validation 

Output validation answers questions on how well 
the simulation results compare with the 
perceived real world. The DoD W&A 
Recommended Practices Guide calls this stage 
“results validation.” Results validation compares 
the responses of the simulation with known or 
expected behavior from the subject it represents 
to ascertain that those responses are sufficiently 
accurate for the range of intended uses of the 
simulation. This process includes comparison of 
simulation outputs with the results of controlled 
tests, sensitivity analyses, or expert opinion. 
One useful approach to output validation is 
graphical comparison. Output validation 
contains the answers to the following questions: 


Is the simulation sensitive to the proper 
input data items; such as, does the difference 
between two sets of simulation results 
reflect a believable result given the variation 
in the input data sets? Is the simulation 
credible? 

Do the individual pieces such as the 
functional areas and system units of the 
simulation adequately represent their 
counterparts in the “real world”? 
(Comparison of models and simulations to 
real systems) 

Is the model and simulation complete and 
are the functions adequately modeled? 
(Completeness and adequacy of actual 
system functions) This may involve 
inspection of design documents to compare 


Does the simulation produce results that are 
plausible? (Veracity and acceptability of the 
results) 

Is the output/result reasonable relative to the 
inputs? Is there mathematical consistency 
between input data and results? For 
example, we may expect greater detection 
range results from raising the power in a 
radar. 

Does the difference in input produce the 
expected proportional change in the output? 
Do changes in data result in anticipated and 
comparative changes in output? 

How does the model output compare to 
historical data, test data, laboratory data or 
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exercise data? Compare simulation to “real 
world” data. For example, how well does a 
simulated ballistic missile trajectory 
correlate with actual flight test data? 

Graphical Comparison 

Graphical comparison is a subjective, inelegant, 
and heuristic, yet quite practical approach to 
output validation. The graphs of values of model 
variables over time are compared with the graphs 
of values of system variables to investigate 
characteristics such as similarities in 
periodicities, skewness, number and location of 
inflection points, logarithmic rise and linearity, 
phase shift, trend lines, and exponential growth 
constants. The graphs in Figure 6.3.1-1 
demonstrate how graphical comparison can be 
used to evaluate correlation between model 
variables and system variables. 





Figure 6.3.1-1 Examples of Graphical Comparisons 
between Model and System Variables 


6.4 Methods of Validation 

Procedural and technical approaches that are 

frequently used in validation are described 

below. These methods may be used as 

appropriate to support the validation effort. 

• Peer review. A validation approach that 
involves conducting critical and detailed 
examinations of internal representations of 
data inputs, key parameters, and resulting 
output by personnel who are knowledgeable 
about modeling the functional areas 
represented in simulation. 

• Independent review. Validation performed 
by competent, objective reviewers who are 
independent of the simulation developer. It 
may consist of examination of the adequacy 
and completeness of the verification and 
validation methods already performed by the 
simulation developer. 

• Face validation. The validation process of 
determining whether a model, on the 
surface, seems reasonable to personnel who 
are knowledgeable about the system or 
phenomena under study. 

• Comparison to other models and 

simulations. This uses results or output 
from internal algorithms or other 

simulations already accredited for use in 
similar applications as part of both structural 
and output validation. Direct comparison of 
code, documentation, input data, and results 
are the primary techniques used. 

• Piecewise Validation or functional 
decomposition. Decomposing the model 
into functional components is often a great 
aid in the validation process. Functional 
area SME for each part of the model and 
simulation are brought in to examine in 
detail the documentation, code, and output 
to determine the validity of each segment of 
the decomposed model. 

• Stress tests and sensitivity analysis. SME 

validates whether the model and simulation 
provides proper output responses to inputs 
for the entire spectrum of valid input data. 

• Animation and graphics playback. These 

techniques allow the analyst to see the 
simulation’s behavior through time. 

• Turing tests. These tests ask experts in 
operation of a system to differentiate 
between data flow, controls and outputs of 
the real world system and modeling and 
simulation results. 
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The method for achieving the desired level of 
validation is Model-Test-Model (M-T-M). M-T- 
M is a method that uses test and evaluation 
results in an iterative method of successive 
simulation improvement, with each successive 
step increasing the overall validity. The M-T-M 
process is accomplished through the following 
steps: model the scenario; observe test play; 
constrain the model to test conditions; compare 
the measures to observations; adjust the 
simulation; re-run the model and repeat the cycle 
as necessary. The basic components of M-T-M 
are: pretest modeling, measures and test 
observations comparison, and posttest modeling. 
These phases are run successively until the 
desired degree of validation is achieved. 

7.0 Accreditation 

Accreditation is the official determination by the 
application sponsor that the capabilities of the 
model and simulation fit the intended use and 
that its limitations will not interfere in drawing 
the correct conclusions. Accreditation occurs at 
a key point in the process to solve a given 
problem. At this point, the person responsible 
for accepting the solution determines the model 
or simulation is sufficient for its intended use. 
Accreditation is a decision to use a model or 
simulation for a specific application (i.e., project 
or program). The decision is supported by as 
much information as is necessary to be credible. 
According to DoD Directive 5000.59, 
accreditation is "the official certification that a 
model or simulation is acceptable for a specific 
purpose." Accreditation, then, must be 

associated with a specific purpose or application. 

This section describes the process leading up to 
and supporting accreditation. This process is 
shown in Figure 5.0-1. Note that the 
accreditation process is conducted concurrently 
with the V&V process. For EADTB, V&V is a 
part of the accreditation process. 



FIGURE 7.0-1 THE ACCREDITATION 
PROCESS 


7.1 Accreditation Requirements 

The accreditation process begins with the 
determination of accreditation requirements. 
These requirements include the V&V 
requirements as well as other M&S 
characteristics needed and constraints based on 
application limitations. The V&V requirements 
are determined by the application sponsor. 
These key functions are prioritized in order of 
importance to the application. The V&V status 
of each of the key functions is then determined. 
The V&V status reflects whether V&V has been 
performed on this M&S function, the quality of 
the V&V performed, and the actual V&V 
findings. Other accreditation requirements 
include M&S characteristics that can affect the 
decision for the model’s or simulation's approval 
and use. These factors include (a) model or 
simulation development and use history, (b) 
operational environment requirements, (c) 
configuration management status, (d) 
documentation status, and (e) other known 
capabilities and limitations of the model or 
simulation and supporting data bases. 

7.2 Accreditation Planning 

The application-specific accreditation 
requirements are satisfied based on the 
accreditation plan. The plan contains the list of 
requirements to be satisfied, the method of 
meeting each requirement, the agent responsible 
for each requirement, the overall resources 
needed, and the schedule for satisfying the 
requirements. A major subset of the 
accreditation plan is the V&V plan. For 
EADTB, this is a separate plan because it is the 
major work to be accomplished. Each 
requirement is examined, and the optimum 
method of requirement satisfaction is selected. 
The optimum is based on a trade-off of cost. 
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resources, and time to complete. Each 
requirement satisfaction method then is grouped 
appropriately and integrated to give an overall 
approach to meeting the requirements. 
Requirements that drive the cost, resources, or 
schedule are re-examined to find more efficient 
ways of satisfying them. If no alternative can be 
found for a requirement that is excessively costly 
or time consuming, it is reconsidered. Based on 
its priority, the requirement can be accepted as 
is, reformulated to make it easier to accomplish, 
or eliminated. Once the methods for all 
requirements are accepted, an integrated resource 
list and schedule is developed. If the V&V 
requirements are to be accomplished through a 
separate plan, they are documented separately. 
The approach to meeting all requirements is 
documented in the accreditation plan. 

7.3 Accreditation Plan Execution 

Once the accreditation plan has been approved, 
satisfaction of the requirements begins. The 
non-V&V requirements are met using the 
methods specified in the accreditation plan. 
These methods usually involve identifying 
sources of and collecting information, which 
should be documented. If execution of the 
accreditation plan is long or detailed, interim 
reports and reviews of progress may be 
appropriate. 

7.4 Acceptability Assessment 

The acceptability assessment reviews all 
accreditation information, both V&V and non- 
V& V, and develops a list of capability voids, 
weaknesses, and mismatches of model or 
simulation functions and characteristics versus 
application acceptability criteria. If 

modifications to the model or its data base are 
necessary to fill voids or correct weaknesses, 
approaches to these modifications along with the 
resources required and a schedule are developed 
and documented. If the voids or weaknesses can 
be avoided by limiting the uses of specific 
models or algorithms, these limitations are 
documented. If there is a potential, yet 
undetermined weakness because of a lack of 
V&V, the additional V&V needed to determine 
if the weakness exists is estimated in terms of 
resources and time. The capability voids and 
weaknesses are analyzed together to develop an 
overall recommendation for EADTB use, 
EADTB use with limitations, EADTB 
modifications, additional V&V, or EADTB 


rejection. The results of the acceptability 
assessment and the recommendation with its 
rationale are documented in the acceptability 
assessment report and briefed to the accreditation 
authority. 

Accreditation Authority 

The accreditation authority then has the 
responsibility to review the results of the 
acceptability assessment and, based on that 
information as well as other factors, make a 
decision. Among the other factors the 
accreditation authority may consider are a 
projected program schedule slip (for an 
acquisition program) or an anticipated budget 
decrease (or increase). The accreditation 
authority may ask the acceptability assessment 
team to develop additional information or 
different approaches to fill voids or eliminate 
weaknesses in a model's or simulation's 
capabilities before a decision is made. The 
decision can be one or a combination of the 
following: 

a. Use the EADTB as it is for the application. 

b. Use the EADTB with limitations in that 
use. 

c. Modify the EADTB before use. 

d. Perform additional V&V. 

e. Do not use the EADTB for this 
application. Alternatives C through E 
incur additional costs and cause schedule 
changes. Alternative E is the most severe 
because it causes the process to begin 
again at developing the M&S. 

8.0 W&A Documentation 

A crucial step in the EADTB W&A process is 
the documentation of all V&V findings. These 
documents are used as the primary source 
documents for accreditation. All V&V 
documentation is assembled in the W&A 
library at the IV&V contractor’s facility. These 
documents include the general EADTB W&A 
plan, W&A plans specific to particular tests or 
studies, IV&V analysis reports covering analysis 
of critical algorithms and processes within 
EADTB, IV&V briefings, results of exercises 
and studies, formal W&A reports associated 
with particular studies, the EADTB SSR 
validation plans, certification letters from the 
project offices and system project offices 
(SPOs), all user documentation and all developer 
documentation. Access to information in the 
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W&A library is controlled by the Testbed 
Product Office (TPO). 

A W&A database has been created and 
maintained and is used to track the progress of 
the activities performed related to testing system 
requirements. The database provides traceability 
to the W&A testing approach. The database 
contains a list of the requirements tested and a 
summary of the testing activities performed. 
Critical activities for each requirement are 
identified, performed, and documented in the 
database. Standard results reports are generated 
from the database and are added to a test report. 
The database was created using Microsoft 
Access. 

AH W&A documentation is controlled under 
standard configuration management procedures 
and is updated as needed, to parallel the on-going 
EADTB development. 

9.0 Conclusions 

The result of a sound W&A process as applied 
to EADTB provides confidence in the use of the 
simulation. Risk is reduced in decision-making 
based on results of the simulation and usability is 
increased for future applications of EADTB. 
Costs are contained by performing 
comprehensive W&A and most importantly 
DOD policy requirements are met while 
preserving technical merit. 
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